Vbert #339
Draft
Vbert #339
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
QuentinJGMace
force-pushed
the
vbert
branch
from
e77142f to
5c11cd3
Compare
* modeling * update modeling * update token id default * init files * remove vllama + update torch lower bound for cpu * back to normal transformer bound * clean * Update colpali_engine/models/__init__.py --------- Co-authored-by: QuentinJGMace <[email protected]>
mlconti1
approved these changes
Oct 21, 2025
Comment on lines
-15
to
+18
| ColQwen2_5Omni, | ||
| ColQwen2_5OmniProcessor, | ||
| # ColQwen2_5Omni, | ||
| # ColQwen2_5OmniProcessor, |
There was a problem hiding this comment.
Add comment to the README if ColQwen 2.5 Omni is not supported anymore
|
|
||
| # Process queries. | ||
| queries = [self.processor.query_prefix + q + self.processor.query_augmentation_token * 10 for q in queries] | ||
| # queries = [self.processor.query_prefix + q + self.processor.query_augmentation_token * 10 for q in queries] |
There was a problem hiding this comment.
remove commented lines if not useful
There was a problem hiding this comment.
actually usefull, in modernvbert self.processor.query_prefix is "" but it is useful if somebody wants to reproduce other older models.
Thanks for flagging it out !
| # Process queries. | ||
| queries = [self.processor.query_prefix + q + self.processor.query_augmentation_token * 10 for q in queries] | ||
| # queries = [self.processor.query_prefix + q + self.processor.query_augmentation_token * 10 for q in queries] | ||
| queries = [q + self.processor.query_augmentation_token * 10 for q in queries] if is_str else queries |
There was a problem hiding this comment.
put 10 into a constant (e.g. N_AUGMENTATION_TOKENS)
Comment on lines
+126
to
+127
| else: | ||
| proc_batch[k] = v |
| query_outputs = model(input_ids=inputs["query_input_ids"], attention_mask=inputs["query_attention_mask"]) | ||
| query_outputs = model(**{k[6:]: v for k, v in inputs.items() if k.startswith("query")}) | ||
| # feed only kwargs with 'doc_' prefix | ||
| doc_outputs = model(**{k[4:]: v for k, v in inputs.items() if k.startswith("doc")}) |
There was a problem hiding this comment.
define var/constant for len("doc:")
| """ | ||
| Helper function to reshape negative doc inputs to (batch_size * num_neg_docs, ...) | ||
| """ | ||
| neg_doc_inputs = {k[8:]: v for k, v in inputs.items() if k.startswith("neg_doc")} |
There was a problem hiding this comment.
define var/constant for 8
There was a problem hiding this comment.
could rename variables for more clarity and use constants, and add doc
| assert scores.shape == (len(ds), len(ds)), f"Expected shape {(len(ds), len(ds))}, got {scores.shape}" | ||
|
|
||
| # # Check if the maximum scores per row are in the diagonal of the matrix score | ||
| # assert (scores.argmax(dim=1) == torch.arange(len(ds), device=scores.device)).all() |
There was a problem hiding this comment.
why is this commented out?
This comment was marked as resolved.
This comment was marked as resolved.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Still to do:
Modify all negatives loss, not just the ones used for vbert